MongoDB Atlas VectorDB [clean] #2996

ranfysvalle02 · 2024-06-21T23:35:05Z

Why are these changes needed?

MongoDB has been ranked as the best vector database(https://www.mongodb.com/blog/post/atlas-vector-search-commands-highest-developer-nps-retool-state-ai-2023-survey) in the Retool AI report, so it is quite important to add MongoDB vector search as an option for Autogen RAG.

You can easily start the MongoDB vector search on a free tier M0 MongoDB Atlas cluster. Free tier cluster provides the full functionality of the MongoDB vector search. https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/

But why is MongoDB such a standout? Well, there are a few key reasons.

MongoDB Atlas integrates smoothly with existing databases. For organizations already using MongoDB, this means a seamless expansion into the vector storage—no major system overhauls required!
MongoDB Atlas is built to handle operational heavy-lifting. It excels when serving large-scale, mission-critical applications, offering robustness and reliability where it counts.
MongoDB's flexibility in handling a variety of data types and structures makes it perfectly suited to the complexity of vector embeddings.

As such, implementing MongoDB as a Retrieval Agent can unlock new potential in your AI applications, bringing the full power of vector storage to bear.

Related issue number: 711

Closes #711

Checks

I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

First steps towards MongoDB as a VectorDB.

Co-authored-by: Jib <Jibzade@gmail.com>

update PREDEFINED_VECTOR_DB and change name to MongoDBAtlasVectorDB; upsert=True update logic; no more index/collection name check.

Co-authored-by: Jib <Jibzade@gmail.com>

with MongoDB Atlas Vector Search indexes, things work a little differently than traditional MongoDB indexes. Atlas Search indexes are separate entities managed by the Atlas Search service. Deleting a collection doesn't automatically remove the associated Atlas Search index - leading to errors

Co-authored-by: Jib <Jibzade@gmail.com>

Co-authored-by: Li Jiang <bnujli@gmail.com>

Update agentchat_mongodb_RetrieveChat.ipynb Update agentchat_mongodb_RetrieveChat.ipynb

ranfysvalle02 · 2024-06-21T23:37:30Z

@Hk669 @thinkall made a fresh pull request, with cleaner commit history. I did a lot of "learning" on that last pull request :)

I think we are pretty close to getting MongoDB into Autogen

autogen/agentchat/contrib/vectordb/mongodb.py

test/agentchat/contrib/vectordb/test_mongodb.py

thinkall · 2024-06-22T00:38:04Z

Test is still skipped:

https://github.com/microsoft/autogen/actions/runs/9621000866/job/26540852406?pr=2996#step:11:26

Need to update contrib-tests.yml

name: Install mongodb
run: |
pip install -e .[retrievechat-mongodb]

codecov-commenter · 2024-06-22T02:40:48Z

Codecov Report

Attention: Patch coverage is 0.92593% with 107 lines in your changes missing coverage. Please review.

Project coverage is 26.01%. Comparing base (89c2f20) to head (5f89f21).
Report is 3 commits behind head on main.

Files	Patch %	Lines
autogen/agentchat/contrib/vectordb/mongodb.py	0.00%	104 Missing ⚠️
autogen/agentchat/contrib/vectordb/base.py	25.00%	3 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (89c2f20) and HEAD (5f89f21). Click for more details.

HEAD has 27 uploads more than BASE
| Flag | BASE (89c2f20) | HEAD (5f89f21) | |------|------|------| |unittests|1|28|

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2996      +/-   ##
==========================================
- Coverage   32.49%   26.01%   -6.49%     
==========================================
  Files          93      100       +7     
  Lines       10097    10299     +202     
  Branches     2167     2356     +189     
==========================================
- Hits         3281     2679     -602     
- Misses       6532     7318     +786     
- Partials      284      302      +18

Flag	Coverage Δ
unittest	`12.28% <0.00%> (?)`
unittests	`25.21% <0.92%> (-7.28%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Co-authored-by: Li Jiang <bnujli@gmail.com>

gitguardian · 2024-06-22T03:53:57Z

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
-		MongoDB Credentials	`54655e8`	notebook/agentchat_mongodb_RetrieveChat.ipynb	View secret
-		MongoDB Credentials	`3122301`	notebook/agentchat_mongodb_RetrieveChat.ipynb	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

ranfysvalle02 · 2024-06-22T04:09:37Z

@thinkall - I think there is something going on with testing retrieval?

test/test_retrieve_utils.py ............s.                               [ 56%]
test/agentchat/contrib/retrievechat/test_pgvector_retrievechat.py s      [ 60%]
test/agentchat/contrib/retrievechat/test_qdrant_retrievechat.py s..      [ 72%]
test/agentchat/contrib/retrievechat/test_retrievechat.py s.              [ 80%]
test/agentchat/contrib/vectordb/test_mongodb.py s                        [ 88%]



---------- coverage: platform linux, python 3.10.14-final-0 ----------
Coverage XML written to file coverage.xml


======================== 20 passed, 5 skipped in 52.41s ========================

ranfysvalle02 · 2024-06-22T04:13:51Z

I polluted this PR :( sorry -- lets try this one last time

thinkall · 2024-06-22T10:30:20Z

I polluted this PR :( sorry -- lets try this one last time

There is no need to worry about the commit history. Make a new PR will lost the track history.

ranfysvalle02 and others added 30 commits June 11, 2024 22:23

Create mongodb.py

3024021

MongoDB Class for agentchat

37454dc

First steps towards MongoDB as a VectorDB.

Update mongodb.py

fb86b8a

Update base.py

275315d

Merge branch 'microsoft:main' into main

0bfdc73

Update mongodb.py

5972fd0

Merge branch 'main' into main

0122481

Update autogen/agentchat/contrib/vectordb/mongodb.py

19dbbb8

Co-authored-by: Jib <Jibzade@gmail.com>

Update autogen/agentchat/contrib/vectordb/mongodb.py

0151b3c

Co-authored-by: Jib <Jibzade@gmail.com>

Update autogen/agentchat/contrib/vectordb/mongodb.py

4289ef7

Co-authored-by: Jib <Jibzade@gmail.com>

Update autogen/agentchat/contrib/vectordb/mongodb.py

9d1dbe8

Co-authored-by: Jib <Jibzade@gmail.com>

Update autogen/agentchat/contrib/vectordb/mongodb.py

9a91bc0

Co-authored-by: Jib <Jibzade@gmail.com>

Update autogen/agentchat/contrib/vectordb/mongodb.py

5629700

Co-authored-by: Jib <Jibzade@gmail.com>

Update autogen/agentchat/contrib/vectordb/mongodb.py

fd24baa

Co-authored-by: Jib <Jibzade@gmail.com>

update from PR comments

8f4bb1f

update PREDEFINED_VECTOR_DB and change name to MongoDBAtlasVectorDB; upsert=True update logic; no more index/collection name check.

start notebook progress

61919b2

Update agentchat_mongodb_RetrieveChat.ipynb

524aa9c

Update autogen/agentchat/contrib/vectordb/mongodb.py

85db1af

Co-authored-by: Jib <Jibzade@gmail.com>

Update autogen/agentchat/contrib/vectordb/mongodb.py

217c66a

Co-authored-by: Jib <Jibzade@gmail.com>

Update mongodb.py

9f0cdf1

Merge branch 'main' into main

9c760b6

Create test_mongodb.py

f537b24

Update autogen/agentchat/contrib/vectordb/mongodb.py

fe94978

Co-authored-by: Jib <Jibzade@gmail.com>

Merge branch 'main' into main

0c08b31

Merge branch 'main' into main

145e53f

Merge branch 'main' into main

10b7d50

Getting closer...

a9d33e0

pre-commit fix

2ce1cd1

Update agentchat_mongodb_RetrieveChat.ipynb

86a94ac

thinkall and others added 14 commits June 21, 2024 20:44

Merge branch 'main' into main

c64e679

Update agentchat_mongodb_RetrieveChat.ipynb

a82fe0e

Merge branch 'main' of https://github.com/ranfysvalle02/autogen

db47b8f

Update mongodb.py

9a32f0d

Update notebook/agentchat_mongodb_RetrieveChat.ipynb

081e24e

Co-authored-by: Li Jiang <bnujli@gmail.com>

Update contrib-tests.yml

48913ab

testing is getting closer!

4097b4c

Update mongodb.py

841f2d9

Update agentchat_mongodb_RetrieveChat.ipynb Update agentchat_mongodb_RetrieveChat.ipynb

pre-commit run --show-diff-on-failure --color=always --all-files

3ea97f2

Update mongodb.py

634f2f8

Merge branch 'main' into main

1ca4d70

Merge branch 'main' into main

4690f32

Merge branch 'main' into main

9a7fbdd

clean commit history

0be4fcd

ranfysvalle02 mentioned this pull request Jun 21, 2024

MongoDB + new abstraction of vectordb #2942

Closed

3 tasks

thinkall reviewed Jun 22, 2024

View reviewed changes

autogen/agentchat/contrib/vectordb/mongodb.py Outdated Show resolved Hide resolved

test/agentchat/contrib/vectordb/test_mongodb.py Outdated Show resolved Hide resolved

test/agentchat/contrib/vectordb/test_mongodb.py Outdated Show resolved Hide resolved

Merge branch 'main' into main

5f89f21

ranfysvalle02 and others added 2 commits June 21, 2024 23:30

Update test/agentchat/contrib/vectordb/test_mongodb.py

d99247a

Co-authored-by: Li Jiang <bnujli@gmail.com>

Merge branch 'main' of https://github.com/ranfysvalle02/autogen

069141e

Fabian Valle added 2 commits June 21, 2024 23:56

Update mongodb.py

2a30319

Update test_mongodb.py

093272c

ranfysvalle02 closed this Jun 22, 2024

ranfysvalle02 mentioned this pull request Jun 22, 2024

+mdb atlas vectordb [clean_final] #3000

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MongoDB Atlas VectorDB [clean] #2996

MongoDB Atlas VectorDB [clean] #2996

ranfysvalle02 commented Jun 21, 2024

ranfysvalle02 commented Jun 21, 2024

thinkall commented Jun 22, 2024 •

edited

Loading

codecov-commenter commented Jun 22, 2024 •

edited

Loading

gitguardian bot commented Jun 22, 2024 •

edited

Loading

ranfysvalle02 commented Jun 22, 2024

ranfysvalle02 commented Jun 22, 2024

thinkall commented Jun 22, 2024

MongoDB Atlas VectorDB [clean] #2996

MongoDB Atlas VectorDB [clean] #2996

Conversation

ranfysvalle02 commented Jun 21, 2024

Why are these changes needed?

Related issue number: 711

Checks

ranfysvalle02 commented Jun 21, 2024

thinkall commented Jun 22, 2024 • edited Loading

codecov-commenter commented Jun 22, 2024 • edited Loading

Codecov Report

gitguardian bot commented Jun 22, 2024 • edited Loading

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

ranfysvalle02 commented Jun 22, 2024

ranfysvalle02 commented Jun 22, 2024

thinkall commented Jun 22, 2024

thinkall commented Jun 22, 2024 •

edited

Loading

codecov-commenter commented Jun 22, 2024 •

edited

Loading

gitguardian bot commented Jun 22, 2024 •

edited

Loading